As shipped, Debian Jessie (8.0) did not include kernel support for the USB controller on APM X-Gene based machines like the Mustang. In fact, at the time of writing this that support has not yet gone upstream into the mainline Linux kernel either but patches have been posted by Mark Langsdorf from Red Hat. This means that installing Debian is more awkward than it could be on these machines. They don't have optical drives fitted normally, so the neat isohybrid CD images that we have made in Debian so far won't work very well at all. Booting via UEFI from a USB stick will work, but then the installer won't be able to read from the USB stick at all and you're stuck. :-( The best way so far for installing Debian is to do a network installation using tftp etc. Well, until now... :-) I've patched the Debian Jessie kernel, then re-built the installer and a netinst image to use them. I've put a copy of that image up at http://cdimage.debian.org/cdimage/unofficial/arm64-mustang/ with more instructions on how to use it. I'm just submitting the patch for inclusion into the Jessie stable kernel, hopefully ready to go into the 8.1 point release.

It was quite an experience to partake in the Jessie release and also a rather long Saturday . This post is mostly a time line of how I spent my release day with doing the actual release. I have glossed over some details the post is long enough without these. :) We started out at 8 (UTC) with a final dinstall run, which took nearly 2 hours. It was going to take longer, but we decided to skip the synchronisation to coccia.debian.org (the server hosting the DD-accessible mirror of release.debian.org). The release itself started with the FTP masters renaming the aliases of Squeeze, Wheezy and Jessie to oldoldstable, oldstable and stable respectively. While they worked, the release team reviewed and double checked their work. After an hour (~11), the FTP masters reported that the stable releases were ready for the final review and the SRMs signed the relevant Release files. Then the FTP masters pushed the stable releases to our CD build server, where Steve McIntyre started building the installation images. While Steve started with the CDs, the FTP masters and the release team continued with creating a suite for Stretch. On the FTP/release side, we finished shortly before 12:30. At this point, our last ETA from Steve suggested that the installation media would take another 11 and a half hours to complete. We could have opened for mirror synchronisation then, but we decided to wait for the installation media. At 12:30, there was a long intermission for the release team in the release process. That was an excellent time to improve some of our tools, but that is for another post. :) We slowly started to resume around 22:20, where we tried to figure out when to open for the mirror synchronisation to time it with the installation media. We agreed to start the mirror sync at 23:00 despite the installation media not being completely done then. They followed half an hour later, when Steve reported that the last CD was complete. At this point, all that was left was to update the website and send out the press announcement. Sadly, we were hit by some (minor) issues then. First, I had underestimated the work involved in updating the website. Secondly, we had no one online at the time to trigger an out of band rebuild of the website. Steve and I spent an hour and a half solving website issues (like arm64 and ppc64el not being listed as a part of the release). Unsurprisingly, I decided to expand our the release checklist to be slightly more verbose on this particular topic. My Saturday had passed its 16th hour, when I thought we had fixed all the website issues (of course, I would be wrong) and we would now just be waiting for the an automatic rebuild. I was tempted to just punt it and go to bed, when Paul Wise rejoined us at about 01:25. He quickly got up to speed and offered to take care of the rest. An offer I thankfully accepted and I checked out 15 minutes later at 01:40 UTC. That more or less covers the Jessie release day from my PoV. After a bit of reflection inside the release team, we have found several points where we can improve the process. This part certainly deserves its own post as well, which will also give us some time to flesh out some of the ideas a bit more. :)
Filed under: Debian, Release-Team

I'm happy with the progress we've made for debian-installer and related packages for the Jessie release. We're going to end up with a release that's better in a number of ways than what we've had before. 1. Big EFI enhancements I've already blogged a lot about the stuff I've worked on here, so I'll just summarise for now some of the improvements we've got over Wheezy.

1. A fix for systems that (badly) dual-boot in EFI and BIOS mode such that after installing Debian you wouldn't get a sensible choice of which OS to boot (#763127).
2. A workaround for broken EFI implementations: an option to install the grub-efi bootloader to the removable media path in case the system firmware does not load grub-efi from the correctly registered boot path. (#746662).
3. Addition of 32-bit EFI to our i386 installation images, to support both some older systems and some brand new systems that need it. This has unfortunately stopped those i386 images from working on some of the oldest Intel-based Apple Mac machines, so we've added an extra Mac-only flavour of i386 netinst without EFI in case people need it.
4. Significantly better support for Intel-based Apple Macs in general, to the point that installing Debian on lots of these machines should now be much easier and doesn't depend on extra third-party software such as rEFIt or rEFInd. I've massively updated the Debian wiki page at https://wiki.debian.org/MacMiniIntel with more details for specific models of Mac Mini. I'm hoping to provide similarly updated information for Mac laptops too - see below!
   Massive thanks to the lovely folks at Mythic Beasts for providing me with a range of machines to test with here!
5. Support for mixed-mode EFI systems like the Intel Bay Trail: a 64-bit platform crippled with a 32-bit EFI firmware. I believe Jessie will be the first release of a Linux distribution to support these machines fully!

2. Openstack images In collaboration with Thomas Goirand, we now have amd64 Openstack Jessie image builds being produced every week, and there will be an official image made to go with the Jessie release too. See http://cdimage.debian.org/cdimage/openstack/testing/ for the current image. 3. Debian-live images As of a few weeks ago, we've also added started doing weekly builds of live Debian images for amd64 and i386, using software and configuration from the Live Systems Project. See http://cdimage.debian.org/cdimage/weekly-live-builds/ for the current weekly images. These will be produced in sync with the Jessie release too. 4. New architectures We've added installation media for the two new architectures added in Jessie: arm64 and ppc64el. I'm particularly proud of the arm64 images. With help from Ian Campbell, Leif Lindholm and Thomas Schmitt I've managed to make EFI-compatible CD images in an isohybrid design that means they should also work when copied directly to a USB stick. Hopefully this will help this new platform to become just as easy to install as any x86 PC is today. Hopefully post-Jessie we'll even be able to start providing live images and openstack images for more architectures too. More help needed yet! First of all, we're planning to release Jessie as Debian 8 this coming Saturday (25th April). Help with testing the installation and live images as they're produced would be lovely - please join us on the #debian-cd channel on irc.debian.org and we'll co-ordinate there. Secondly, there's an almost endless variety of machines out there. I've updated information about how Debian installation works on some of the more awkward Mac Mini machines, but we don't yet cover all the bases even there. It would be great to update the information about other machines such as the Macbook range as well - currently a lot of these pages are well out of date and won't be helpful for new users. Please test on machines if you have them, and help improve Debian's documentation here.

As a follow-up to my post about bootstrapping arm64 in Debian, we've had more hardware given to Debian for us to use in porting and building packages for arm64. Applied Micro sent me an X-Gene development machine to set up and use. Unfortunately, the timing was unlucky and the machine sat on my desk unopened for a few weeks while I was on long holiday in Australia. Once I was back, I connected it up and got it working. Out of the box, a standard Jessie arm64 installation worked using network boot (dhcp and tftp). I ran through d-i as normal and installed a working system, then handed it over to the DSA and buildd folks to get the machine integrated into our systems. Easy! The machine is now up and running as arm-arm-03.debian.org and has been building packages for a few weeks now. You can see the stats here on the buildd.debian.org site. In terms of installation, I also got the machine to boot using one of our netinst images on a USB stick, but that path didn't get very far. The USB drivers for this hardware have only quite recently gone into the mainline kernel, and haven't been backported to the Debian Jessie kernel yet. I'm hoping to get those included shortly. There's also an option to replace the U-Boot firmware that came with the X-Gene with UEFI instead, which would be much more helpful for a server platform like this. I'll look into doing that upgrade soon too, but probably after the Jessie release is done. I don't want to jinx things just now. *grin* Thanks to APM for their generous donation here, and particularly to Richard Zenkert for his help in getting this machine shipped to us.

One final update on my work for UEFI improvements in Jessie! All of my improvements have been committed into the various Debian packages involved, and the latest release candidate for Jessie's debian-installer build (RC2) works just as well as my test builds on the Bay Trail system I've been using (Asus X205TA). Job done! :-) I'm still hoping to maybe get more hardware support for this particular hardware included in Jessie, but I can't promise. The mixed EFI work has also improved things for a lot of Mac users, and I'm planning to write up a more comprehensive list of supported machines in the Debian wiki (for now). There's now no need to use any of the older test installer images - please switch to RC2 for now. See http://cdimage.debian.org/cdimage/jessie_di_rc2/ for the images. If you want to install a 64-bit system with the 32-bit UEFI support, make sure you use the multi-arch amd64/i386 netinst or DVD. Otherwise, any of the standard i386 images should work for a 32-bit only system. Upstreaming My kernel patch to add the new /sys file was accepted upstream a while back, and has been in Linus' master branch for some time. It'll be in 4.0 unless something goes horribly wrong, and as it's such a tiny piece of code it's trivial to backport to anything remotely recent too. I've also just seen that my patch for grub2 to use this new /sys file has been accepted upstream this week. Again, the change is small and self-contained so should be easy to copy across into other trees too. Mixed EFI systems should now have better support across all distros in the near future, I hope.

Jo and I just got back from our massive holiday in Australia. We had an awesome time overall, fitting in lots of stuff in 4 weeks. Time for a quick write-up and some photos! We flew into Sydney, then straight onto Uluru for the obligatory sunset and sunrise viewings. We didn't climb the Rock, both for sensitivity reasons and (to be more honest!) it looked way too much like hard work in 40-plus degree heat. Coach over to Alice Springs, where we had a very quick look around before taking the Ghan train down to Adelaide. The train was fun for a day, and we got to see a lot of desert. In Adelaide, we had a look around the city (lovely colonial feel!) and got a couple of evenings in fun comedy shows at the Fringe. Great fun! On to Tasmania, where we did a quick (3 days) run around the island by car: into Hobart, up the east coast. Stopped in Swansea (a nice version!) for some heavenly Devonshire teas, then on up to Grindelwald near Launceston. Visited Trowunna Wildlife Park to see (and cuddle!) lots of local animals, which was amazing - Jo's favourite day of the holiday. Then on to Queenstown and drive back down to Hobart past some impossibly beautiful views around Cradle Mountain. Tassie's gorgeous - like the best bits of Scotland, Wales and Cornwall but with even fewer people and better weather. Next, on to Sydney for Harry and Cath's wedding. We stayed up in Chatswood. Not knowing anything about the area beforehand, we were a little surprised to basically find ourselves back in Hong Kong! We spent most of the weekend catching up with friends from the wedding group, and the wedding itself was at Quarantine Station, overlooking the harbour. It couldn't have been a more perfect location / weather / view for our friends' big day! We squeezed in a couple of the open-top bus tours of Sydney on the Sunday, but got caught in the horrendous storm that hit and ended up sheltering downstairs under cover on the bus. I'm told Bondi is lovely, but it all looked grey from the bus. :-P Down to Melbourne on the train (bit of a wasted day, in hindsight), where we wandered around the city quite a bit. Caught up with an old friend who lives there for a day, and we did a wine tour up the Yarra Valley which was fun too. Up to Port Douglas, where we headed out to the Reef for my highlight of the holiday: a snorkelling tour with some local marine experts who showed us the local flora and fauna. We also visited a local Aboriginal cultural centre, skyrail and scenic railway around Kuranda village. Down to Hervey Bay and a 1-day tour of Fraser Island - an amazing place in combination with quite a thrill-ride experience just being driven around on the sand tracks. Finally, down to Brisbane where we wandered around and visited both the Lone Pine Koala Sanctuary (more cuddles!) and the Gold Coast. Then the long flights home. Whew! We're knackered now. We knew we could't fit everything in, but we're glad we travelled all over and got tastes of almost everything. Now we can work out where we want to spend more time on our future visit(s). We'll definitely want to head over and see Perth and some of WA next time, and definitely more time in Tasmania, Sydney and Adelaide.

I've been working on this for too long without really talking about it, so let's fix that now! VLANd is a simple (hah!) python program intended to make it easy to manage port-based VLAN setups across multiple switches in a network. It is designed to be vendor-agnostic, with a clean pluggable driver API to allow for a wide range of different switches to be controlled together. There's more information in the README file. I've just released v0.2, with a lot of changes included since the last release:

• Massive numbers of bugfixes and code cleanups
• Improve how we talk to the Cisco switches - disable paging on long output
• Switch from "print" to "logging.foo" for messages, and add logfile support
• Improved test suite coverage, and added core test scripts for the lab environment

I've demonstrated this code today in Hong Kong at the Linaro Connect event, and now I'm going on vacation for 4 weeks. Australia here I come! :-)

Seen on an ATM in Hong Kong airport.

About a year and a half after I started writing the openstack-debian-images package, I m very happy to announce to everyone that, thanks to Steve McIntyre s help, the official OpenStack Debian image is now generated at the same time as the official Debian CD ISO images. If you are a cloud user, if you use OpenStack on a private cloud, or if you are a public cloud operator, then you may want to download the weekly build of the OpenStack image from here: http://cdimage.debian.org/cdimage/openstack/testing/ Note that for the moment, there s only the amd64 arch available, but I don t think this is a problem: so far, I haven t found any public cloud provider offering anything else than Intel 64 bits arch. Maybe this will change over the course of this year, and we will need arm64, but this can be added later on. Now, for later plans: I still have 2 bugs to fix on the openstack-debian-images package (the default 1GB size is now just a bit too small for Jessie, and the script exits with zero in case of error), but nothing that prevents its use right now. I don t think it will be a problem for the release team to accept these small changes before Jessie is out. When generating the image, Steve also wants to generate a sources.tar.gz containing all the source packages that we include on the image. He already has the script (which is used as a hook script when running the build-openstack-debian-image script), and I am planning to add it as a documentation in /usr/share/doc/openstack-debian-images. Last, probably it would be a good idea to install grub-xen, just as Ian Campbell suggested to make it possible for this image to run in AWS or other Xen based clouds. I would need to be able to test this though. If you can contribute with this kind of test, please get in touch. Feel free to play with all of this, and customize your Jessie images if you need to. The script is (on purpose) very small (around 400 lines of shell script) and easy to understand (no function, it s mostly linear from top to bottom of the file), so it is also very easy to hack, plus it has a convenient hook script facility where you can do all sorts of things (copying files, apt-get install stuff, running things in the chroot, etc.). Again, thanks so much to Steve for working on using the script during the CD builds. This feels me with joy that Debian finally has official images for OpenStack.

Time for another update on my work for UEFI improvements in Jessie! I've spent more time on the integration of 32-bit grub-efi with a 64-bit Debian system, and just published a new test image on pettersson. I've added:

• a patch to the Linux kernel to add a new /sys file which exposes the size of the underlying UEFI platform (32- or 64-bit).
• a patch to grub2 to read that new /sys file in grub-install to determine the right version of grub-efi to install by default
• a patch to grub-installer to do similar

These remove the manual steps that were necessary for a 64-bit installation with the previous build. I've just used this exact image (and a network mirror) to install a fully-functional 64-bit Gnome system on the X205TA, simply by selecting "64-bit install" from the GRUB menu and following prompts. Yay! Visit http://cdimage.debian.org/cdimage/unofficial/efi-development/jessie-upload3/ to download and test the image. Now, there's no guarantee that the kernel patch I've submitted to the linux-efi folks will be accepted in its current form, and even if it is I'll have to get it and the other code I've written accepted into the various packages and then into Jessie! But for now this image should work just fine for Bay Trail folks I hope! WARNING: this CD is provided for testing only. Use at your own risk! If you have appropriate (U)EFI hardware, please try this image and let me know how you get on, via the debian-cd and debian-boot mailing lists. For now, I'm going to pause development here. The core code I'm using to make these images is all in the debian-cd and d-i repos, and I'll push the other patches once I know they'll work with the kernel. But I've got a slew of other things that I need to work on in the next few weeks, in no particular order:

• RC bugs filed against abcde
• Sorting out Mac-only 32-bit netinst images (only EFI boot? without EFI?)
• Regular openstack image generation for Jessie
• Regular debian-live image generation for Jessie
• ...

I'm currently not planning to make all of Debian's amd64 images bootable using 32-bit UEFI like this image - I'm happy to leave this as just an option for our multi-arch i386/amd64 images (netinst or DVD only). I think that's a reasonable compromise here, and it's also the easiest thing for me to do with the current debian-cd build system. Finally, apologies if you've asked me questions about the earlier images in this series and I've not responded yet. Fixing that ASAP!

I promised to write about this a long time, ooops... :-) Another ARM port in Debian - yay! arm64 is officially a release architecture for Jessie, aka Debian version 8. That's taken a lot of manual porting and development effort over the last couple of years, and it's also taken a lot of CPU time - there are ~21,000 source packages in Debian Jessie! As is often the case for a brand new architecture like arm64 (or AArch64, to use ARM's own terminology), hardware can be really difficult to get hold of. In time this will cease to be an issue as hardware becomes more commoditised, but in Debian we really struggled to get hold of equipment for a very long time during the early part of the port. First bring-up in Debian Ports To start with, we could use ARM's own AArch64 software models to build the first few packages. This worked, but only very slowly. Then Chen Baozi and the folks running the Tianhe-2 supercomputer project in Guangzhou, China contacted us to offer access to some arm64 hardware, and this is what Wookey used for bootstrapping the new port in the unofficial Debian Ports archive. This has now become the normal way for new architectures to get into Debian. We got most of the archive built in debian-ports this way, and we could then use those results to seed the initial core set of packages in the main Debian archive. Second bring-up - moving into the main Debian archive By the time that first Debian bring-up was done, ARM was starting to produce its own "Juno" development boards, and with the help of my boss^4 James McNiven we managed to acquire a couple of those machines for use as official Debian build machines. The existing machines in China were faster, but for various reasons quite difficult to maintain as official Debian machines. So I set up the Junos as buildds just before going to DebConf in August 2014. They ran very well, and (for dev boards!) were very fast and stable. They built a large chunk of the Debian archive, but as the release freeze for Jessie grew close we weren't quite there. There was a small but persistent backlog of un-built packages that were causing us issues, plus the Juno machines are/were not quite suitable as porter boxes for Debian developers all over the world to use for debugging their packages on the new architecture. More horsepower - Linaro machines This is where Linaro came to our aid. Linaro's goal is to help improve Free and Open Source Software on ARM, and one of the more recent projects in Linaro is a cluster of servers that are made available for software developers to use to get early access to ARMv8 (arm64) hardware. It's a great way for people who are interested in this new architecture to try things out, port their software or indeed just help with the general porting effort. As Debian is seen as such an important part of the FLOSS ecosystem, we managed to negotiate dedicated access to three of the machines in that cluster for Debian's use and we set those up in October, shortly before the freeze for Jessie. Andy Doan spent a lot of his time getting these machines going for us, and then I set up two of them as build machines and one as the porter box we were still needing. With these extra machines available, we quickly caught up with the ever-busy "Needs-Build" queue and we've got sufficient build power now to keep things going for the Jessie release. We were officially added to the list of release architectures at the Cambridge mini-Debconf in November, and all is looking good now! And in the future? I've organised the loan of another arm64 machine from AMD for Debian to use for further porting and/or building. We're also expecting that more and more machines will be coming out soon as vendors move on from prototyping to producing real customer equipment. Once that's happened, more kit will be available and everybody will be able to have arm64-powered computers in the server room, on their desk and even inside their laptop! Mine will be running Debian Jessie... :-) Thanks! There's been a lot of people involved in the Debian arm64 bootstrapping at various stages, so many that I couldn't possibly credit them all! I'll highlight some, though. :-) First of all, Wookey's life has revolved around this port for the last few years, tirelessly porting, fixing and hacking out package builds to get us going. We've had loads of help from other teams in Debian, particularly the massive patience of the DSA folks with getting early machines up and running and the prodding of the ftpmaster, buildd and release teams when we've been grinding our way through ever more package builds and dependency loops. We've also had really good support from toolchain folks in Debian and ARM, fixing bugs as we've found them by stressing new code and new machines. We've had a number of other people helping by filing bugs and posting patches to help us get things built and working. And (last but not least!) thanks to all the folks who've helped us beg and borrow the hardware to make the Debian arm64 port a reality. Rumours of even more ARM ports coming soon are entirely scurrilous... *grin*

Time for another update on my work for UEFI improvements in Jessie! I now have a mixed 32- and 64-bit UEFI netinst up and running right now, which will boot and install on the Asus X205TA machine I have. Since the last build, I've added 64-bit (amd64) support and added CONFIG_EFI_MIXED in the kernel so that the 64-bit kernel will also work with a 32-bit UEFI firmware. Visit http://cdimage.debian.org/cdimage/unofficial/efi-development/jessie-upload2/ to download and test the image. There are a few other missing pieces yet for a complete solution, but I'm getting there...! WARNING: this CD is provided for testing only. Use at your own risk! If you have appropriate (U)EFI hardware, please try this image and let me know how you get on, via the debian-cd and debian-boot mailing lists.

Time for another update on my work for UEFI improvements in Jessie! I've got an i386-only UEFI netinst up and running right now, which will boot and install on the Asus X205TA machine I have. I've got the required i2c modules included in this build so that the installer can use the keyboard and trackpad on the machine - useful...! See #772578 for more details about that. Visit http://cdimage.debian.org/cdimage/unofficial/efi-development/jessie-upload1/ to download and test the image. There are a few other missing pieces of hardware support for this machine yet, but the basics are there. See the wiki for more information. My initial i386 test CD is not yet going to do an amd64 installation for you, but it should let you get going with Debian on these machines! I'm going to continue working on that 32-64 support next. WARNING: this CD is provided for testing only. Use at your own risk! If you try this on an early Intel-based Mac, it will not work. Otherwise, this should likely work for most folks using 32-bit x86 hardware just like any other Debian Jessie daily netinst build. If you have appropriate (U)EFI hardware, please try this image and let me know how you get on, via the debian-cd and debian-boot mailing lists.

A month ago , I wrote about my plans for improved (U)EFI support in Jessie. It's about time I gave an update on progress! I spoke about adding support for installing grub-efi into the removable media path (#746662). That went into Debian's grub packages already, but there were a couple of bugs. First of all, the code could end up prompting people about EFI questions even when they didn't have any grub-efi packages installed. Doh! (#773004). Then there was an unexpected bug with case-insensitive file name handling on FAT/VFAT filesystems (#773092). I've posted (and tested!) patches to fix both, hopefully in an upload any day now. Next, I mentioned getting i386 UEFI support going again. This is a major feature that a lot of people have been asking for. It's also going to involve quite a bit of effort... Our existing (amd64) UEFI-capable images in Debian use the standard x86 El Torito CD boot method, with two boot images provided. One of these images gives us the traditional isolinux BIOS-boot support. The second option is an alternate El Torito image, including at a 64-bit version of grub-efi. For most machines, this works just fine - the BIOS or UEFI firmware will automatically pick the correct image and everybody's happy. This even works on our multi-arch i386/amd64 CDs and DVDs - isolinux will boot either kernel from the first El Torito image, or the alternate UEFI image is amd64 only. However, I can now see that there's been a long-standing issue with those multi-arch images, and it's to do with Macs. On the advice of Matthew Garrett, I've borrowed an old 32-bit Intel Mac to help testing, and it's quite instructive in terms of buggy firmware! The firmware on older 32-bit Intel Macs crashes hard when it detects more than one El Torito boot image, and I've now seen this happen myself. I've not had any bug reports about this, so I can only assume that we haven't had many users try that image. As far as I can tell, they've been using the normal i386 images in BIOS boot mode, and then struggling to get bootloaders working afterwards. There are a number of different posts on the net explaining how to do that. That's OK, but... If I now start adding 32-bit UEFI support to our standard set of i386 images, this will prevent users of old Macs from installing Debian. I could just say "screw it" and decide to not support those users at all, but that's not a very nice thing to do. If we want to continue to support them and add 32-bit UEFI support, I'll have to add another flavour of i386 image, either a "Mac special" or a "32-bit UEFI special". I'm not keen on doing that if I could avoid it, but the two options are mutually exclusive. Given the Mac problem is only on older hardware which (hopefully!) will be dying out, I'll probably pick that one as the special-case CD, and I'll make an extra netinst flavour only for those users to boot off. So, I've started playing with i386 UEFI stuff in the last couple of weeks too. I almost immediately found some showstopper behaviour bugs in the i386 versions of efivar and efibootmgr (#773412 and #773007), but I've debugged these with our Debian maintainer (Jared Dominguez) and the upstream developer (Peter Jones) and fixes should be in Jessie very soon. As I mentioned last month, the machines that most people have been requesting support for are the latest Bay Trail-based laptops and tablets. There are using 64-bit Intel Atom CPUs, but crippled with 32-bit UEFI firmware with no BIOS compatibility mode. This makes for some interesting issues. It's probably impossible to get a true answer why these machines are so broken by design, but there are several rumours. As far as I can see, most of these machines seem to ship with a limited version of 32-bit Windows 8.1. 32-bit Windows is smaller than 64-bit Windows, so fits much better in limited on-board storage space. But 32-bit Windows won't boot from 64-bit UEFI, so the firmware needed buggering to match. Ugh! To support these Bay Trail machines properly, we'll want to add a 32-bit UEFI installation option to our 64-bit images. I can tweak our CDs so that both 32-bit and 64-bit versions of grub-efi are included, and the on-board UEFI will load the right one needed. Then I'll need to make sure that all the 64-bit images also include grub-efi-ia32-bin from now on. With some extra logic, we'll need to remember that these new machines need that package installing instead of grub-efi-amd64-bin. It shouldn't be too hard, but let's see! :-) So, I've been out and bought one of these machines, an Asus X205TA. Lucas agreed that Debian will reimburse me (thanks!), so I'm not stuck with spending my own money on an otherwise unwanted machine! I can see via Google that none of the mainstream Linux distros support the Bay Trail machines fully yet, so there's not a lot of documentation yet. Initial boot on the new machine was easy using a quick-hack i386 UEFI image on USB, but from there everything went downhill quickly. I'll need to investigate some more, but the laptop's own keyboard and trackpad are not detected by the installer system. Neither is its built-in WiFi. Yay! I had to go and dig out a USB hub to connect the installer image USB key, a keyboard, mouse and a USB hard drive to the machine, as it only has 2 USB ports. I've taken a complete backup of the on-board 32GB flash before I start experimenting, so I can restore the machine back to its virgin state for future testing. I guess I now have a project to keep me busy over Christmas...! In other news, we've been continuing work on UEFI support for and within the new arm64 port. My ARM/Linaro colleague Leif Lindholm has been back-porting upstream kernel features and bug fixes to make d-i work, and filing Debian bugs when silly things break on arm64 because people don't think about other architectures (e.g #773311, doh!). As there are more and more people interested in (U)EFI support these days, I've also proposed that we create a new debian-efi mailing list to help focus discussion. See #773327 and follow up there if you think you'd use the list too! You can help! Same as 2 years ago, I'll need help testing some of these images. For the 32-bit UEFI support, I now have some relevant hardware myself, but testing on other machines too will be very important! I'll start pushing unofficial Jessie EFI test images shortly - watch this space.

So, my work for Wheezy gave us working amd64 UEFI installer images. Yay! Except: there were a few bugs that remained, and also places where we could deal better with some of the more crappy UEFI implementations out there. But, things have improve since then and we should be better for Jessie in quite a few ways. First of all, Colin and the other Grub developers have continued working hard and quite a lot of the old bugs in this area look to be fixed. I'm hoping we're not going to see so many "UEFI boot gives me a blank black screen" type of problems now. For those poor unfortunates with Windows 7 on their machines, using BIOS boot despite having UEFI support in their hardware, I've fixed a long-standing bug (#763127) that could leave people with broken systems, unable to dual boot. We've fixed a silly potential permissions bug in how the EFI System Partition is mounted: (#770033). Next up, I'm hoping to add a workaround for some of the broken UEFI implementations, by adding support in our Grub packages (and in d-i) for forcing the installation of a copy of grub-efi in the removable media path. See #746662 for more of the details. It's horrid to be doing this, but it's just about the best thing we can do to support people with broken firmware. Finally, I've been getting lots of requests for adding i386 (32-bit x86) UEFI support in our official images. Back in the Wheezy development cycle, I had test images that worked on i386, but decided not to push that support into the release. There were worries about potentially critical bugs that could be tickled on some hardware, plus there were only very few known i386 UEFI platforms at the time; the risk of damage outweighed the small proportion of users, IMHO. However, I'm now revisiting that decision. The potentially broken machines are now 2 years older, and so less likely to be in use. Also, Intel have released some horrid platform concoction around the Bay Trail CPU: a 64-bit CPU (that really wants a 64-bit kernel), but running a 32-bit UEFI firmware with no BIOS Compatibility Mode. Recent kernels are able to cope with this mess, but at the moment there is no sensible way to install Debian on such a machine. I'm hoping to fix that next (#768461). It's going to be awkward again, needing changes in several places too. You can help! Same as 2 years ago, I'll need help testing some of these images. Particularly for the 32-bit UEFI support, I currently have no relevant hardware myself. That's not going to make it easy... :-/ I'll start pushing unofficial Jessie EFI test images shortly.

I've been crap about blogging lately. Let's see if I can fix that. Back in February and March, Jo and I went on vacation for 2 weeks touring California and Nevada. We had an awesome time and we got to see and do lots of fun stuff. I'm not going to go into all the details, as it's a long time ago now...! I've got a massive set of photos online, though. However, two things struck me as odd when we were there. I'm travelling quite regularly to the US these days due to my work in Linaro, but these still seemed new when I saw them this February/March. These are, admittedly trivial things, but they really stood out for me. Maybe I'm a little weird? :-) Curved shower curtain tracks I guess I'm not the only one who's been annoyed by shower curtains sticking to me in the shower, but I'd not really paid much thought to it until now. Suddenly, as of maybe 18 months ago I'm seeing most hotel bathrooms replacing the straight curtain track with a curved one, to stop that happening. This photo shows that process with both tracks visible... Waterproof/washable TV remotes This one really surprised me. As Jo will attest, I have a little bit of an obsession with TVs and set-top boxes in hotel rooms. This dates from my time working for Amino where we made set-top boxes, and I got into the habit of checking what products were in the hotels I stayed in. I've seen a range of weird and wonderful setups over the years, but never this one before. In two of the hotels on our trip, they had replaced the normal TV remotes with washable/wipe-down ones. Weird...

For the second year running, we held our mini-debconf in Cambridge last weekend. Roughly 70 people turned up this year to the ARM offices in Cambridge over the 4 days. Alongside our originally planned general sprint work on Thursday and Friday, the Release Team had their own sprint to work through remaining post-freeze decisions about policies, architectures, naming etc. The rest of us worked through a range of topics all over Debian: installer, admin, ARM arch support etc. We even managed to fix Andy's laptop :-). Saturday and Sunday were two days devoted to more traditional conference sessions. Our talks covered a wide range of topics: d-i progress and an update from the Release Team, backup software and an arm64 laptop project to name but a few. I was very happy that the Release Team announced Zurg"Stretch" and "Buster" as upcoming release names, and obviously it was lovely to have arm64 and ppc64el confirmed as release architectures for Jessie. It's a shame to see the kfreebsd ports dropped from the official Jessie list, but let's see if the porters can do an unofficial release anyway. I'll help if I can with things like CD builds... Several volunteers from the DebConf video team were on hand too, so our talks were recorded and are already online at http://meetings-archive.debian.net/pub/debian-meetings/2014/mini-debconf-cambridge/webm/. Yay! Again, the mini-conf went well and feedback from attendees was universally positive. We may run again next year. More importantly, I can confirm that we're definitely planning on bidding to host a full DebConf in Cambridge in the summer of 2017. Thanks to all our helpers, and of course to our sponsors: ARM for providing the venue and infrastructure for the event, and Codethink for helping with food costs.

There s a lot of discussion / moaning /arguing at this time, so I thought I d post something about how LAVA got into Debian Jessie, the work involved and the lessons I ve learnt. Hopefully, it will help someone avoid the disappointment of having their package missing the migration into a future stable release. This was going to be a talk at the Minidebconf-uk in Cambridge but I decided to put this out as a permanent blog entry in the hope that it will be a useful reference for the future, not just Jessie. Context LAVA relies on a number of dependencies which were at the time all this started NEW to Debian as well as many others already in Debian. I d been running LAVA using packages on my own system for a few months before the packages were ready for use on the main servers (I never actually installed LAVA using the old virtualenv method on my own systems, except in a VM). I did do quite a lot of this on my own but I also had a team supporting the effort and valuing the benefits of moving to a packaged system. At the time, LAVA was based on Ubuntu (12.04 LTS Precise Pangolin) and a new Ubuntu LTS was close (Trusty Tahr 14.04) but I started work on this in 2013. By the time my packages were ready for general usage, it was winter 2013 and much too close to get anything into Ubuntu in time for Trusty. So I started a local repo using space provided by Linaro. At the same time, I started uploading the dependencies to Debian. json-schema-validator, django-testscenarios and others arrived in April and May 2014. (Trusty was released in April). LAVA arrived in NEW in May, being accepted into unstable at the end of June. LAVA arrived in testing for the first time in July 2014. Upstream development continued apace and a regular monthly upload, with some hotfixes in between, continued until close to the freeze. At this point, note that although upstream is a medium sized team, the Debian packaging also has a team but all the uploads were made by me. I planned ahead. I knew that I would be going to Macau for Linaro Connect in February a critical stage in the finalisation of the packages and the migration of existing instances from the old methods. I knew that I would be on vacation from August through to the end of September 2014 including at least two weeks with absolutely no connectivity of any kind. Right at this time, Django1.7 arrived in experimental with the intent to go into unstable and hence into Jessie. This was a headache for me, I initially sought to delay the migration until after Jessie. However, we discussed it upstream, allocated time within the busy schedule and also sought help from within Debian with the RFH tag. Rapha l Hertzog contributed patches for django1.7 support and we worked on those patches upstream, once I was back from vacation. (The final week of my vacation was a work conference, so we had everyone together at one hacking table.) Still there was more to do, the django1.7 patches allowed the unit tests to complete but broke other parts of the lava-server package and needed subsequent tweaks and fixes. Even with all this, the auto-removal from testing for packages affected by RC bugs in their dependencies became very important to monitor (it still is). It would be useful if some packages had less complex dependency chains (I m looking at you, uwsgi) as the auto-removal also covers build-depends. This led to some more headaches with libmatheval. I m not good with functional programming languages, I did have some exposure to Scheme when working on Gnucash upstream but it wasn t pleasant. The thought of fixing a scheme problem in the test suite of libmatheval was daunting. Again though, asking for help, I found people in the upstream team who wanted to refresh their use of scheme and were able to help out. The fix migrated into testing in October. Just for added complications, lava-server gained a few RC bugs of it s own during that time too fixed upstream but awkward nonetheless. Achievement unlocked So that s how a complex package like lava-server gets into stable. With a lot of help. The main problem with top-level packages like this is the sheer weight of the dependency chain. Something seemingly unrelated (like libmatheval) can seriously derail the migrations. The package doesn t use the matheval support provided by uwsgi. The bug in matheval wasn t in the parts of matheval used by uwsgi. It wasn t in a language I am at all comfortable in fixing but it s my name on the changelog of the NMU. That happened because I asked for help. OK, when django1.7 was scheduled to arrive in Debian unstable and I knew that lava was not ready, I reacted out of fear and anxiety. However, I sought help, help was provided and that help was enough to get upstream to a point where the side-effects of the required changes could be fixed. Maintaining a top-level package in Debian is becoming more like maintaining a core package in Debian and that is a good thing. When your package has a lot of dependencies, those dependencies become part of the maintenance workload of your package. It doesn t matter if those are install time dependencies, build dependencies or reverse dependencies. It doesn t actually matter if the issues in those packages are in languages you would personally wish to be expunged from the archive. It becomes your problem but not yours alone. Debian has a lot of flames right now and Enrico encouraged us to look at what else is actually happening in Debian besides those arguments. Well, on top of all this with lava, I also did what I could to help the arm64 port along and I m very happy that this has been accepted into Jessie as an official release architecture. That s a much bigger story than LAVA yet LAVA was and remains instrumental in how arm64 gained the support in the kernel and various upstreams which allowed patches to be accepted and fixes to be incorporated into Debian packages. So a roll call of helpers who may otherwise not have been recognised via changelogs, in no particular order:

• Steve McIntyre (Debian & Linaro)
• Rapha l Hertzog (Debian)
• Dave Pigott (Linaro)
• R mi Duraffort (Linaro)
• Sjoerd Simons (Debian)
• Antonio Terceiro (Debian and formerly Linaro)
• Martin Pitt (Debian)
• Jordi Mallach (Debian)
• Hector Oron (Debian)
• Colin Watson (Debian)

Also general thanks to the Debian FTP and Release teams. Lessons learnt

1. Allow time! None of the deadlines or timings involved in this entire process were hidden or unexpected. NEW always takes a finite but fairly lengthy amount of time but that was the only timeframe with any amount of uncertainty. That is actually a benefit it reminds you that this entire process is going to take a significant amount of time and the only loser if you try to rush it is going to be you and your package. Plan for the time and be sceptical about how much time is actually required.
2. Ask for help! Everyone in Debian is a volunteer. Yes, the upstream for this project is a team of developers paid to work on this code (and largely only this code) but the upstream also has priorities, requirements, objectives and deadlines. It s no good expecting upstream to do everything. It s no good leaving upstream insufficient time to fit the required work into the existing upstream schedules. So ask for help within upstream and within Debian ask for help wherever you can. You don t know who may be able to help you until you ask. Be clear when asking for help how would someone test their proposed fix? Exactly what are you asking for help doing? (Hint: everything is not a good answer.)
3. Keep on top of announcements and changes. The release team in Debian have made the timetable strict and have published regular updates, guidelines and status notes. As maintainer, it is your responsibility to keep up with those changes and make others in the upstream team aware of the changes and the implications. Upstream will rely on you to provide accurate information about these requirements. This is almost more important than actually providing the uploads or fixes. Without keeping people informed, even asking for help can turn out to be counter-productive. Communicate within Debian too talk to the teams, send status updates to bugs (even if the status is tag 123456 + help).
4. Be realistic! Life happens around us, things change, personal timetables get torn up. Time for voluntary activity can appear and disappear (it tends to disappear far more often than extends, so take that into account too).
5. Do not expect others to do the work for you asking for help is one thing, leaving the work to others is quite another. No complaining to the release team that they are blocking your work and avoid pleading or arguing when a decision is made. The policies and procedures within Debian are generally clear and there are quite enough arguments without adding more. Read the policies, read the guidelines, watch how other packages and other maintainers are handled and avoid those mistakes. Make it easy for others to help deliver what you want.
6. Get to know your dependency chain follow the links on the packages.debian.org pages and get a handle on which packages are relevant to your package. Subscribe to the bug pages for some of the more high-risk packages. There are tools to help. rc-alert can help you spot problems with runtime dependencies (you do have your own package installed on a system running unstable if not, get that running NOW). Watching build-dependencies is more difficult, especially build-dependencies of a runtime dependency, so watch the RC bug lists for packages in your dependency chain.

Above all else, remember why you and upstream want the packages in Debian in the first place. Debian is a respected distribution and has an acknowledged reputation for stability and portability. The very qualities that you and your upstream desire from having your package in Debian have direct implications for the amount of work and the amount of time that will be required to get your packages into Debian and keep them there. Having your package in Debian will bring considerable benefits but you will be required to invest a considerable amount of time. It is this contribution which is valuable to Debian and it is this work which will deliver the benefits you seek. Being an expert in the one package is wildly inadequate. Debian is about the system, the whole distribution and sooner or later, you as the maintainer will be absolutely required to handle something which is so far out of your comfort zone it s untrue. The reality is that you are not expected to fix that problem you are expected to handle that problem and that includes seeking and acknowledging the help of others. The story isn t over until release day. Having your package in testing the day before the freeze is one step. It may be a large step, but it is only one. The status of that package still needs monitoring. That long dependency chain can still come back and bite. Don t wait for problems to surprise you. Finally One thing I do ask is that other upstream teams and maintainers think about the dependency chain they are creating. It may sound nice to have bindings for every interpreted language possible when releasing your compiled library but it does not help people using that library. Yes, it is more work releasing the bindings separately because a stable API is going to be needed to allow version 1.2.3 to work alongside 1.2.2 and 1.3.0 or the entire effort is pointless. Consider how your upstream package migrates. Consider how adding yet another build-dependency for an optional component makes things exponentially harder for those who need to rely on that upstream. If it is truly optional, release it separately and keep backwards compatibility on each side. It is more work but in reality, all that is happening is that the work is being transferred from the distribution (where it helps only that one distribution and causes duplication into other distributions) into the upstream (where it helps all distributions). Think carefully about what constitutes core functionality and release the rest separately. Combining bindings for php, ruby, python, java, lua and xslt into a single upstream release tarball is a complete nonsense. It simply means that the package gets blocked from new uploads by the constant churn of being involved in every transition that occurs in the distribution. There is a very real risk that the package will miss a stable release simply by having fingers in too many pies. That hurts not only this upstream but every upstream trying to use any part of your code. Every developer likes to think that people are using and benefiting from their effort. It s not nice to directly harm the interests of other developers trying to use your code. It is not enough for the binary packages to be discrete migrations happen by source package and the released tarball needs to not include the optional bindings. It must be this way because it is the source package which determines whether version 1.2.3 of plugin foo can work with version 1.2.0 of the library as well as with version 1.3.0. Maintainers regularly deal with these issues so talk to your upstream teams and explain why this is important to that particular team. Help other maintainers use your code and help make it easier to make a stable release of Debian. The quicker the freeze & release process becomes, the quicker new upstream versions can be uploaded and backported.

It's past time I wrote about how Linaro's students fared in this year's Google Summer of Code. You might remember me posting earlier in the year when we welcomed our students. We started with 3 student projects at the beginning of the summer. One of the students unfortunately didn't work out, but the other two were hugely successful. Gaurav Minocha was a graduate student at the University of British Columbia, Vancouver, Canada. He worked on Linux Flattened Device Tree Self-checking, mentored by Grant Likely from Linaro's Office of the CTO. Gaurav achieved all of his project's goals, and he was invited to Linaro's recent Linaro Connect USAConnect conference in California to meet people and and talk about his project. He and Grant presented a session on their work; it was filmed, and video is online. Grant said he was very happy with Gaurav's "strong, solid performance" during the project. Varad Gautam was a student at Birla Institute of Technology and Science, Pilani, India. He succeeded in porting UEFI to the BeagleBone Black. Leif Lindholm from the Linaro Enterprise Group was his mentor for the summer. At the end of the summer, Varad delivered a UEFI port ready for booting Linux and his code was included in Linaro's September UEFI release. Leif said that he was "very pleased with Varad's self sufficiency and ability to pick up an entirely new software project very quickly". We were hoping to invite Gaurad to Connect in California also, but travel document delays got in the way. With luck we'll see him at the next Connect in Hong Kong in February 2015. Well done, guys! It was great to work with these young developers for the summer, and we wish them lots more success in their future endeavours. Google have also just confirmed that they will be running the Summer of Code program again in 2015. I'm hoping that Linaro will be accepted again next year as a mentoring organisation. I'll post more about that early next year.

I find it somewhat interesting that thousands of years later that our society still uses Publilius Syrus sententiae though I imagine the tendency to leave well enough alone means such phrases stay in usage.

One weekend Steve McIntyre asked me if I could find a source of some of some 40mm fans for some systems with some pretty strict requirements. They needed to be long life and shift a lot of air to combat a persistent overheating issue.

I sat with him and went through the Farnell utterly hateful parametric web interface and eventually came up with a couple of options which were very expensive. Only then did I stop and ask what the actual problem was.

Steve showed me one of the Debian ARM buildd boxes which are Marvell development machines. These systems are powerful quad core machines housed in compact steel enclosures.

There is a single 40mm fan trying to provide cooling for the entire enclosure. When the units are placed horizontally and used intermittently this proves adequate. Unfortunately when the system are arranged vertically in a rack and run at full load continuously they often overheat and have to be restarted. In addition the small high speed fans need replacing frequently as their bearings wore out quickly.

This was obviously causing some issues for the ARM Debian ports which Steve wanted to rectify. After talking the problem through for a while we came to the conclusion we could use much larger 60mm fans to blow air directly through the top of the case onto the cpu heatsink.

Larger fans can be run much more slowly to move a similar volume of air to the smaller 40mm fans which gives a much longer service life.

Steve proceeded to order enough parts to allow us to modify all the Debian systems, this worked out cheaper than a single "special" 40mm high volume fan.

I acquired a rather large steel hole punch, I chose this tool because it produces a much superior finish to a hole cutter and this project demanded a high level of finish (not to mention I loved having a valid excuse to own and use a huge allen key!)

If we had simply been modifying a single case I would have measured and marked up by hand. With the prospect of altering at least eight I laser cut a template from plywood which Andy Simpkins took great glee in excessively annotating.

We also used the opportunity to add bolt holes to securely attach the 2.5 inch SATA drives instead of using sticky pads.

Steve and I modified a single system to begin with both to check our alignment and the efficacy of the change. We were pleasantly surprised to discover that hoiby could now repeatedly do kernel compiles with all four cores flat out which was not possible before. The measured CPU temperature, which had previously been around 90 C, did not rise above 40 C

Steve, Andy and I then arranged a day where we took all the remaining units out of the rack at ARM, modified and returned them. We used the facilities at the Cambridge Makespace where I am a member to do the modifications.

I broke two 3mm drill bits and dulled a 4mm bit drilling all the holes, Roger Smith was good enough to loan us the use of his "Christmas tree bit" to ream the fan hole out to 16mm so we could thread the hole punch and cut the 60mm fan aperture out.

We managed to get quite an assembly line going and, in my opinion, the results look pretty professional.

It has been several months since we did this work and these systems continue to run without issue. To complete the story we can see some graphs courtesy of the DSA munin instance.

You can clearly see the huge drop in temperature at the end of Week 25 despite the continuously high CPU load. Also there is only a single gap in the data after the changes (these indicate crashes where data was not recorded) where before there were frequent and extensive times where the systems were simply unusable.

One reason I continue to enjoy Debian so much is the wide variety of ways in which I can contribute not only by maintaining my packages. Sometimes this kind of work does not receive the credit it deserves and hopefully highlights a small part of the frantic paddling that goes on under the serene surface of the Debian project to keep things "just working".